hi i'm richard juday i'm the
robotic vision manager in the tracking
and communications division here at the
johnson space center uh today we're in
the hybrid vision laboratory where we do
a combination of vision techniques we
use laser lights we use spatialite
modulators
and we use a thing that we'll talk about
some more today which is
a image remapping technique that allows
machine vision
to have some of the aspects that have
proven advantageous in the development
of robot of human vision and other
biological human systems
the main function of the programmable
remapper within the machine vision
environment
is to provide things like rotation and
variance and scale in variants and you
have some of those features built into
your
vision system that you're watching the
show with
you know what my face looks like as i
stand here as i look as i stand closer
or closer or closer
you don't have to remember a different
shape of face
because you have a a character to your
vision system
that
doesn't mind the changes of scale
machines have more trouble with that and
we built the programmable remapper in
order to assist with that kind of a
problem in machine vision now some of
the robotic vision
applications that the agency has
uh would include working with
a spacecraft in orbit we have machines
that simulate some of the
change in viewing angle and
and distance that
will be required in space operations
beyond this kind of an application
we also intend to build vision systems
that will land robotically on mars
i am very turned on to the idea that i
might be able to tell my grandkids that
i worked on a vision system that enabled
a robot to land on the surface of mars
technology utilization's interest in
this in this particular technology is
for its application to
some human low vision problems that's
one of the things that the agency is
supposed to do is develop technology
that can be spun off into
public application
some of the things that can help make a
machine see better
we think have a solid chance
of helping the
person who has certain forms
of diminished vision
make the best use of the remaining
function that he has it's not a cure
it's not available next week but we
think that
within something on the order of five
years
this kind of technology the results of
our research here
uh can have a benefit to the
low vision community within the within
the the country
uh i have formed an association here
with uh dr david lotion
from the university of houston's college
of optometry
dave came and sought out what technology
there might be at the
johnson space center that
would assist him in his activities
and this is the association that
resulted
hello i'm david lotion associate
professor and assistant dean at the
university of houston college of
optometry
dr richard today and i have developed a
collaboration trying to bring some of
this nasa technology developed using the
remap programmable remapper to a low
vision application
there are many problems that individuals
have with low vision
these problems usually are associated
with visual field defects that is
regions of the world that they cannot
see
these can be
mainly isolated to the central vision as
in the case of age-related maculopathy
or in peripheral vision as in the case
of retinitis pigmentosa these
individuals have problems with specific
tasks including reading
facial recognition
object recognition mobility and driving
let me give you some examples of these
this is the back of the eye or the
retina this individual has age-related
maculopathy as you notice the central
portion of the retina is affected by the
disease process
individuals with this disease
have difficulty
reading print because wherever they look
they will see a region that is missing
or black
these individuals
these individuals will also have a
problem with facial recognition here you
can see the full face a face we all
recognize but these
individuals may see something like this
all the detail of the face is missing
in the case of a peripheral field defect
as as here shown with retinitis
pigmentosa again this is the retina or
the back of the eye
individuals with this disease
will not see a full
view of the world but rather a very
small portion
or isolated part of the world
what we're trying to do with the
programmable remapper
is take information that would normally
fall within these defects and bring it
into
the viable retina
let me give you some examples of image
image warping
some of these remappings we can show you
applications with a grid here you see a
reduced field of a grid
normally someone with a peripheral field
defect would be given a a device that
minifies the entire field this would
allow more information to go into the
central field where they have their
healthy retina
with remapping we can take the
information
and
actually
show more information in the central
field or a larger portion larger in the
central field and gradually go off into
the periphery and here we can get the
central field exactly the same size as
our reduced field but we had more
information in the periphery of the
field one of the problems with
overall minification
is that people lose localization within
the field and they have a hard time
doing any kind of mobility that is
walking
in the case of central field defect
the information with remapping is pulled
outside of the central defect you can
see here a line coming down will
actually be remapped around
the field effect
notice that we don't have this
distortion throughout the field but just
around the defect we don't have to
remap the entire field instead we can
only do part of it
here is a partial remapping where we're
not taking all the information that
would fall behind the defect but just
part of that information
now this can be applied to
application of reading
as you can see here information is lost
behind the defect and it'll be difficult
to read across the line
by remapping we can take information
that would normally fall with behind
this defect and remap it outside there
is slight distortion but the letters are
still legible
for example to eat you we cannot see
this here but to eat you up the word up
is pulled out of the defect
this would make it easier we believe for
individuals to read text
we can also apply this to facial
recognition
individuals with the central defects
would see we can pull the information
out of the defect you can see here what
happens when we do a complete remapping
we get a lot of distortion
however with a partial remapping we can
get features outside of defect that may
indeed help individuals recognize faces
well let's talk about what we might be
able to do with some
field remapping with the machine that we
have here dave
suppose that a normal person's field of
view
were
about that size uh
then uh what might a person with uh with
the pigmentosa
retinitis pigmentosa that you were
mentioning what might they see well
richard obviously has a lot of forms of
disease but they would be left with a
central region
of of good vision and this disease
progresses any information that would
fall out in the periphery here would
actually be lost
they just wouldn't be able to see that
they wouldn't be able to see that they'd
have a very restricted field of view uh
that looks pretty severe that might be
like looking through a
paper towel tube
uh
well the the obvious thing to do is uh
just use a telescope turn it around
backwards and shrink the whole the whole
field of view into that
that's what's done right now for the
optical aid that we give individuals the
one problem with that is that first of
all they lose a lot of the resolution
but making things so small they can't
pick out objects in the field secondly
there's problem with localization when
you reduce everything and you put your
hand into the field it's hard to tell
where your hand is relative relative to
objects uh and i think if i were walking
down the hall looking through a paper
towel too i might bump into things too
that's true you have a lot of difficulty
with mobility all right so uh what what
i think i hear you saying is that in the
center you'd like to
maintain a fairly high resolution of the
object that we're looking at but bring
some of these
picture elements that uh that used to be
outside the
uh the field of view because of the
defect and then let them jam up a little
more closely together
so that we we have uh picture elements
spaced widely here but but close
together and the periphery of the
remaining part would would that work i
think it would work
obviously when you jam up your your
elements here you're going to get some
distortion but again it won't make that
much difference about the distortion in
that you could tell there's an object
out there whether you can actually tell
what the object is is not as important
as being able to navigate around it okay
so the motion cues that you normally
have if say as you walk down the hall
and things move by you in your
perspective uh you can see that they're
moving without being able necessarily to
read the sign or tell what it is
so we might be able to retain some of
that i think so yes all right well let's
go give it a try i think i know how to
do that
what we have here is the programmable
remapper itself
in the box underneath the monitor it was
manufactured at texas instruments to
specifications laid out here at the
johnson space center
some of the design is there some of it
is ours it functions by taking a video
image as from this camera
and that that video signal is fed into
the remapper
their
coefficients that have been calculated
offline and are
entered into the remapper by
floppy disks
those coefficients are used to push
picture elements around
and to create a display in which a
warping has occurred
now after that image has been warped
in this case it is brought back and
displayed here to a person who
is wearing this uh
helmet that
was manufactured at the
at the university of houston
the
displays here show
what the remapped image is and uh they
are shown to the left and to the right
eyes the same image is uh i will see
here
as dave will see on the monitor
and
so we'll now try on the lightweight
summer version of the
darth vader helmet should point out also
that this version that we have built
here is mainly for testing this is not
going to be the low vision device that
ultimately this will be used with
the
monitor or we have to set up a system
like this so that we could do the
testing to see how useful and how
feasible this type of remapping will be
for patients with low vision
all right so what transform do you have
put in here dave what are we looking for
this is a full field we really haven't
done any transform at all actually we're
just taking the input and
displaying it as an output we're not
changing the transport we're not
changing the image at all
okay so here we are looking at some
letters on a chart
this is actually not remapped at all
richard there what we're taking is the
direct input into the into the remapper
and it's coming out as as it goes in
uh this would be
simulation simulating a full field okay
so i should regard this as my normal
field of view as i look around the
laboratory that's right and quite fine
to see what we got somebody over there
we can
i think this might work dave
uh all right so this is what i might see
with the
reduced field of view from
the retinitis pigmentosa yes and you can
try to look around the room and see how
difficult it is to find objects within
the room
i can't see as much of that computer
scene
nor
can i see near so much
of
chuck and that computer over here
okay well that that would be sort of
hard to find things
all right and what is it that
you might be able to do for us here with
uh remapping
this is the remapped image as i said
this has quite a bit of remapping on it
we have a very sharp fall off
just to give you one example
all right so if i want to read the words
i can put them in the center
but i don't lose them as they go off to
the edge as quickly as i did the other
time it's
sort of like looking at a at a beach
ball
and uh if i go looking around the lab
i can now see the
computer that does not have someone
working at it
and let's go look at a chuck over here
well i can i can see chuck and i can see
the computer
or and i can resolve the uh the computer
if i want to
by putting it here at the center
or i can
resolve chuck here as well
and right here i have the computer and
chuck in the same field and
i think that find that a little easier
to
to use in looking around the uh
the room and trying to find all of that
well this is all very well for
for me to be looking at
i have reasonably good vision but
how are we going to find out if this is
any good for people who have a real
problem well that's our next approach
richard is we're going to
try to have patients try looking at
through this device we're going to
solicit patients from both the
university of houston college of
optometry and the lighthouse of houston
these patients will have several
different
low vision problems and we're going to
try to see how
much better they can see the world with
this particular device
well it's uh
i'm looking forward to that as a look
around here this actually seems rather
easy to get used to
it'll be nice to get some
practical experience with it as we
stated before also
this device will not be the final device
this has to be reduced in size and
obviously cosmesis in order for low fish
low vision patients to want to wear
something like it
some of the
technology that we
are developing with the remapper has
also been explored further for us by
transitions research corporation they
have built some robotic equipment that
use the same kind of technology
our objective here is to have a machine
have some of the same kind of advantages
in pattern recognition
that your own
eye brain combination does in pattern
recognition now you're accustomed to
looking at an object that would appear
like so if you were looking at the front
of a space shuttle
but actually what goes on in the back of
your head is that this
uh this representation
as you see think of it out in front of
your of your head
really looks more like this where the
there's more area uh
this corresponds to small radius near
near the center these these uh fins are
these locations the this is what's
called a log polar domain where the
logarithm of the radius is in this
direction and the
azimuthal angle
is in
the vertical direction
so uh you see that
there is more area in the brain uh given
to the the center of the object now the
advantage for pattern recognition is
that uh as one
were to take uh this object and were to
rotate it or to change change size on it
that uh as appears uh here in
this lower diagram
that its representation here uh in the
log polar domain is merely a translation
of what one sees in the
uh
in the upper picture so you have you
don't have to remember and learn a
really different representation of the
object just that it has slid over and
slid up compared with the
method in which you learned it
originally
now in another simplistic
application of this technology
a centering
algorithm is very easy to implement if
in some of our space applications
you need to
track an object
keep a camera system pointed right
toward the toward that object then
in this as a
cartesian representation of that object
you can see that in order to center this
object you'd push on it here and let it
uh protrude further on on on this side
of these circles well these uh these red
and blue circles uh if you were to take
them over here into the
log
polar uh representation
uh become
this vertical straight line that's red
and this vertical straight line which is
blue
and you see that in order to center the
object the expression of that action
over here becomes find the place where
the object sticks out the farthest and
in in a purely
left-right sense and push to the
left on that
that protrusion which will have this
matching protrusion come out when your
two highest protrusions are
the right distance on this chart which
is about like so separated from each
other
then that object is
is centered
and that's the algorithm that is
implemented in the version of the
tracking that we will shortly show you
okay what should be happening now is
that the
robot is looking for
the
place from which the
uh
docking target bears just the right kind
of resemblance to
what it has been told to look for
uh when it finds that place
uh
it will align itself to be on the axis
of the
uh of the docking target
all all done visually now as a
machine vision process
and we'll then uh dock with
the uh
with that docking target
okay it's looking for the places where
the bulges that result from
the fact that the
uh
box is not a sphere but is instead a
cylinder where those
bulges appear at the right orientation
in its field of view
it should be very close to that by now
okay
what we have here is
the video the same as you would expect
to see it with uh with your eyes
of the video camera looking at the
docking target
in the top you have the
representation of that object as the
robot sees it and this looks very much
like what the part of the what what this
image would look like in
in the back of your head in the in the
brain now we'll give the thing a bit of
a offset just to make the problem harder
and we will set we will tell the
object tell the tracking system
now just to to go dock with the uh
with this target
okay well here we'll we'll start the
demonstration
the
robot now centered the uh
the stocking target and you can see that
it's staying in the center i'm gonna
peel off the uh this overlay that we use
just to make this a little more
easy for us to see
now what what we are seeing is this is
the
uh what we call a cartesian
representation of this object it's the
television image
in much the same
form as you're accustomed to looking at
it yourself but what's happening in the
back of your head is more nearly like
what we see in the
upper region
where this is the uh
the log polar
uh version of this uh uh of this lower
image
now what the robot is doing is it's
because this is a cylinder uh it has
bumps on it compared with uh what it
would look like if it were
a
sphere or just a circle
and it has found a place where those
bumps are in just the orientation that
it wants
at about this angle now it's going to
lower if you see that it's uh
it's jumping it's lowering until the uh
the bumps
have mostly disappeared
and now it's coming in uh on a straight
on approach to the to the docking cart
and so you're looking at
the nose of the spacecraft
we made it